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In the Claims; 

The following listing of claims will replace all prior versions and/or listings of claims in 
the application: 
Listing of Claims: 

What is claimed is: 

1 . (Original) A video surveillance system comprising: 

a sensing unit capable of being operated in a first mode and second mode; and 
a computer system coupled to the sensing unit, the computer system receiving and 
processing image data from the sensing unit, detecting and tracking targets, and determining 
whether the sensing unit operates in the first mode or in the second mode based on the detection 
and tracking of targets. 

2. (Original) The video surveillance system of Claim 1, wherein the sensing unit comprises 
means for improving quality of information obtained about a target. 

3. (Original) The video surveillance system of Claim 2, wherein the means for improving 
quality of information obtained performs at least one of: panning, tilting, and zooming; using a 
super-resolution algorithm; instructing the sensing unit to digitally zoom; employing different 
lenses or filters to view the target; and using a robotic actuator. 
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4. (Original) The video surveillance system of Claim 3, wherein the computer system includes 
a response module adapted to, in the second mode, obtain improved quality information about 
the target. 

5. (Original) The video surveillance system of Claim 2, wherein the computer system provides 
signals to control the means for improving quality of information obtained about a target. 

6. (Original) The video surveillance system of Claim 1, wherein the sensing unit comprises at 
least one of a camera, an infra-red sensor, a thermal sensor, a radio sensor, and an algorithmically 
enhanced camera group. 

7. (Original) The video surveillance system of Claim 1, wherein the computer system 
comprises: 

a vision module to process output of the sensing unit; 
an inference module to process output of the vision module; and 
a response module to perform one or more actions based on the output of the inference 
module. 

8. (Original) The video surveillance system of Claim 7, wherein, in the first mode, the vision 
module detects at least one of blobs and targets. 



-3- 



Applicant: EGNAL et al. 
Appl. No. 10/705,896 



9. (Original) The video surveillance system of Claim 7, wherein, in the first mode, the vision 
module comprises: 

a segmentation module to separate background pixels from foreground pixels; 

a blobizer to receive the foreground pixels from the segmentation module and to group 
connected foreground pixels into coherent blobs; 

a target tracker to process the coherent blobs, determine when they are targets, and to 
obtain position information for each target; 

a classifier to determine a target type for each target; and 

a primitive generation module to generate summary statistics to be sent to the inference 
module. 

10. (Original) The video surveillance system of Claim 7, wherein, in the first mode, the 
inference module determines when at least one specified condition has been either met or 
violated. 

1 1 . (Original) The video surveillance system of Claim 7, wherein the response module is 
adapted to perform at least one of the following: sending an e-mail alert; sounding an audio 
alarm; providing a visual alarm; logging an alert to a persistent medium; making a telephone call; 
transmitting a wireless message to a personal digital assistant; enabling a mechanical actuator; 
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and providing position information to another sensing unit. 

12. (Original) The video surveillance system of Claim 7, wherein, in the second mode, the vision 
module employs at least one of target motion, color and edge information derived from the image 
data to determine target location. 

13. (Original) The video surveillance system of Claim 7, wherein, in the second mode, the vision 
module comprises: 

a motion detection module that determines camera motion and segments target motion 
after having factored out camera motion; 

a color histogram module that calculates a color histogram for a region near each pixel in 
an area near a target; 

an edge detection module that determines edges in any given image; 

a template matching module that uses motion segmentation and color information, as well 
as an edge structure of a target template, to determine a best guess of the target's position; and 

a primitive generation module that generates summary statistical information based on the 
best guess of the target* s position. 

14. (Original) The video surveillance system of Claim 13, wherein the motion detection module 
comprises: 



-5- 



Applicant: EGNALetal. 
Appl. No. 10/705,896 

a comer detection module to find interesting points; 

a search module to find matches for interesting points between successive images; 
a confidence value assignment module to assign confidence values to the matches of the 
interesting points; 

a robust averaging module to determine an estimate from a set of high confidence 
matches; 

a warping module to warp one successive image to another successive image for direct 
comparison of the images; and 

a subtraction module, which subtracts the warped image firom a current image to 
determine which pixels have moved. 

15. (Original) The video surveillance system of Claim 7, wherein, in the second mode, the 
inference module determines whether to switch back to the first mode by monitoring at least one 
of how long the system has been in the second mode, the position of the target in the image, the 
position of the target in a world coordinate fi'ame, and one or more extemal stimuli. 

16. (Original) The video surveillance system of Claim 7, wherein, in the second mode, the 
response module receives higher quality information about the target than in the first mode. 

17. (Original) The video surveillance system according to Claim 1, further comprising: 
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a second sensing unit capable of being operated in the first mode and second mode; and 
a second computer system coupled to the second sensing unit and to the computer system, 
the second computer system receiving and processing image data from the second sensing unit, 
detecting and tracking targets, and determining whether the second sensing imit operates in the 
first mode or in the second mode based on the detection and tracking of targets, 

wherein the computer system and the second computer system cooperate to coordinate 
handoff between the sensing unit and the second sensing unit to track at least one target. 

18. (Original) A method of operating a video surveillance system, the video surveillance system 
including at least one sensing unit capable of being operated in first and second modes, the 
method comprising the steps of: 

operating a sensing unit in the first mode to scan for targets; 

processing image data fi"om the sensing unit in a first mode to detect the presence of an 
interesting target; 

upon detecting an interesting target, operating the sensing unit in the second mode to 
track the interesting target and to improve the quality of information about the interesting target 
over the information that can be obtained in the first mode; and 

processing image data firom the sensing unit in a second mode to track the target by 
sending at least one of pan, tilt, and zoom commands to the sensing unit. 
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19. (Original) The method of Claim 18, wherein the step of operating the sensing unit in the first 
mode comprises the step of performing at least one of: 

operating the sensing unit in a wide-angle mode to scan for targets. 

20. (Original) The method of Claim 18, wherein the step of operating the sensing unit in the 
second mode comprises the steps of: 

obtaining higher quality imagery of a target than that obtained by operating the sensing 
unit in the first mode through the use of at least one of the following: 
panning, tilting, and zooming; 
using super-resolution algorithms; 
digitally zooming into the image; 
employing different lenses or filters; and 
using a robotic actuator. 

21. (Original) The method of Claim 18, wherein the step of processing image data fi-om the 
sensing unit in a first mode comprises the steps of: 

processing the image data with a vision module to detect and locate at least one target; 

and 

determining if at least one predetermined condition has been violated by at least one 

target. 
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22. (Original) The method of Claim 21, wherein the step of processing the image data with a 
vision module comprises the step of: 

geo-locating the at least one object in 3D space. 

23. (Original) The method of Claim 21, wherein the step of processing the image data with a 
vision module comprises the steps of: 

classifying pixels in the image information as background pixels or foreground pixels; 

and 

using the foreground pixels to determine at least one blob. 

24. (Original) The method of Claim 23, further comprising the step of tracking at least one 
possible target based on at least one blob. 

25. (Original) The method of Claim 24, wherein the step of tracking comprises the steps of: 

determining when at least one blob merges or splits into one or more possible targets; and 
performing at least one of: 

filtering and predicting the image location of at least one of the possible targets; and 
calculating a 3D position of at least one of the possible targets if calibration information 
is available. 
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26. (Original) The method of Claim 24, further comprising the step of classifying at least one 
possible target by target type into categories. 

27. (Original) The method of Claim 24, further comprising the step of providing summary 
statistics to aid in the step of determining if at least one predetermined condition has been 
violated by at least one object. 

28. (Original) The method of Claim 27, wherein the summary statistics include at least one of 
the following: an estimate of the target's position in image space; the target's position in a world 
coordinate frame; the target's size in image space; the target's size in the world coordinate frame; 
the target's velocity in image space; the target's velocity in the world coordinate frame; the 
target's trajectory; the target's color; the color of at least one subcomponent of the target; a 
bounding box for the target in an image; and at least one estimate of location and size of at least 
one body part, in the case where the target is human. 

29. (Original) The method of Claim 18, wherein the step of processing image data from the 
sensing unit in a second mode, if the second mode involves camera motion, comprises the steps 
of: 

segmenting the target from the background by detecting motion of the sensing unit and 
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factoring it out; 

calculating color histogram information in a region of at least some pixels of image data 
from the sensing unit; 

calculating an edge image; and 

using template matching, to integrate the results of the steps of motion segmentation, 
edge detection and color histogram formation, to obtain an estimate of the target's position. 

30. (Original) The method of Claim 29, wherein the step of processing image data from the 
sensing unit in a second mode fiirther comprises the step of: 

generating summary statistics based on various features of the target. 

31. (Original) The method of Claim 30, wherein the summary statistics include at least one of 
the following: an estimate of the target's position in image space; the target's position in a world 
coordinate frame; the target's size in image space; the target's size in the world coordinate frame; 
the target's velocity in image space; the target's velocity in the world coordinate frame; the 
target's trajectory; the target's color; the color of at least one subcomponent of the target; a 
bounding box for the target in an image; and at least one estimate of location and size of at least 
one body part, in the case where the target is human. 
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32. (Original) The method of Claim 29, wherein the step of processing image data from the 
sensing unit in a second mode further comprises the steps of: 

monitoring a length of time in which the system has been in the second mode; 

monitoring the various summary statistics to detect if any target characteristic has met a 
predetermined rule for switching to the first mode; 

monitoring for any external stimuli indicating that the system should switch to the first 
mode; and 

if any predetermined conditions are met in any of the monitoring processes, switching 
back to the first mode. 

33. (Original) The method of Claim 18, further comprising the steps of: 

determining a best shot of the target while operating the visual module in either of the 
first and second modes; and 

delivering the best shot back to the user through the response module. 

34. (Original) The method of Claim 18, further comprising the steps of: 

if the target exits the field of view of the sensing unit, directing a second sensing unit to 
scan for and track the target. 

35. (Original) The method of Claim 34, wherein the step of directing a second sensing unit to 
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scan for and track the target comprises the steps of: 

receiving, by the second sensing unit, information about a last known position of the 
target from the sensing unit; 

operating the second sensing unit in the first mode to scan for the target, focusing on a 
region near the last known position; 

processing image data from the second sensing unit in the first mode to detect the 
presence of a target; 

upon detecting a target, operating the second sensing unit in the second mode to track the 
target; and 

processing image data from the second sensing unit in the second mode to track the target 
by sending at least one of pan, tilt, and zoom commands to the second sensing unit, 

wherein, if the target is not detected within a predetermined time period in the step of 
operating the second sensing unit in the first mode to scan for the target, focusing on a region 
near the last known position, the second sensing unit switches to scanning for targets in its entire 
field of coverage. 

36. (Original) A computer-readable medium containing software implementing the method of 
Claim 18. 

37. (Original) A video surveillance system, comprising: 
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at least one sensing unit; 

at least one computer system; and 

the computer-readable medium of Claim 36. 

38. (Original) The method of Claim 18, wherein, if operating the sensing unit in the second 
mode involves camera motion, processing image data from the sensing unit in a second mode 
comprises: 

(a) performing comer detection to find interesting points; 

(b) searching for matches for interesting points in successive images; 

(c) assigning confidence values to the matches of the interesting points; 

(d) employing a robust averaging method to determine an estimate from a subset of the 
matches determined to have high confidence values; 

(e) warping a second successive image towards a first successive image; and 

(f) subtracting the warped image from a current image to determine which pixels have 

moved. 

39. (Original) The method of Claim 38, wherein processing image data from the sensing unit in 
a second mode further comprises: 

performing steps (a)-(f) on reduced resolution images; and 

performing steps (a)-(f) on full resolution images, utilizing the results of performing steps 
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(a)-(f) on reduced resolution images to reduce the computational burden by limiting the searching 
to only a subset of the full resolution images. 

40. (Original) The method of Claim 38, wherein processing image data from the sensing unit in 
a second mode further comprises: 

using three-frame differencing to alleviate false motion. 

41. (Original) A motion detection module to determine camera motion, comprising: 

a comer detection module to find interesting points; 

a search module to find matches for interesting points between successive images; 
a confidence value assignment module to assign confidence values to the matches of the 
interesting points; 

a robust averaging module to determine an estimate from a set of high confidence 
matches; 

a warping module to warp one successive image to another successive image for direct 
comparison of the images; and 

a subtraction module, which subtracts the warped image from a current image to 
determine which pixels have moved. 

42. (Original) The motion detection module of Claim 41, wherein each module operates first on 



-15- 



Applicant: EGNALetal. 
Appl. No. 10/705,896 

reduced resolution image data and then on full resolution image data, using the results of 
operating on reduced resolution image data to reduce a computational burden. 

43. (Original) The motion detection module of Claim 41, wherein the subtraction module uses 
three-frame differencing to alleviate false motions. 

44. (Original) A video-based security system, comprising the video surveillance system 
according to Claim 1. 

45. (Original) A video-based security system, comprising the video surveillance system 
according to Claim 37. 

46. (Original) A method of operating a video-based security system, comprising the method 
according to Claim 18. 

47. (Original) An experiment monitoring system, comprising the video surveillance system 
according to Claim 1. 

48. (Original) An experiment monitoring system, comprising the video surveillance system 
according to Claim 37. 
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49. (Original) A method of monitoring an experiment, comprising the method according to 
Claim 18. 

50. (Original) The method according to Claim 49, further comprising: 

focusing in on at least one predetermined behavior of at least one subject of the 
experiment. 

51. (Original) A sporting event monitoring system, comprising the video surveillance system 
according to Claim 1. 

52. (Original) A sporting event monitoring system, comprising the video surveillance system 
according to Claim 37. 

53. (Original) A method of monitoring a sporting event, comprising the method according to 
Claim 18. 

54. (Original) The method according to Claim 53, further comprising: 

detecting at least one predetermined behavior of a participant in the sporting event. 
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55. (Original) A marketing information gathering system, comprising the video surveillance 
system according to Claim 1. 

56. (Original) A marketing information gathering system, comprising the video surveillance 
system according to Claim 37. 

57. (Original) A method of gathering marketing information, comprising the method according 
to Claim 18. 

58. (Original) The method according to Claim 57, further comprising: 

monitoring at least one behavior of at least one subject. 

59. (Original) The method according to Claim 58, wherein said monitoring comprises: 

detecting interest in a given product. 

60. (Original) The method according to Claim 59, wherein said detecting comprises: 

detecting when a customer reaches for said given product. 

61. (New) An apparatus for video surveillance comprising: 

an apparatus adapted to be coupled to a sensing unit capable of being operated in a 
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first mode and second mode, the apparatus adapted to receive and process image data from the 
sensing unit, detect and track targets, and determine whether the sensing unit operates in the first 
mode or in the second mode based on the detection and tracking of targets. 

62. (New) The apparatus of claim 61, wherein the apparatus comprises appUcation specific 
hardware to emulate a computer and/or software, wherein said hardware is adapted to perform 
said video surveillance. 
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